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EXECUTIVE  SUMMARY 


Because  traditional  decision  research  programs  have  not  been  very  successful  in  addressing 
operational  needs,  the  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 
approximately  10  years  ago  began  to  sponsor  an  alternative  line  of  investigation  into  Naturalistic 
Decision  Making  (NDM).  The  objective  of  this  report  is  to  show  the  value  of  NDM  for  helping  the  Army 
address  current  challenges,  including  its  use  of  information  technologies,  its  need  to  downsize  forces,  and 
a  change  in  its  expected  missions. 

The  goal  of  NDM  research  is  to  examine  the  way  people  make  decisions  under  operational 
conditions.  The  focus  is  on  field  studies  rather  than  on  laboratory  settings  where  naive  subjects  perform 
artificial  tasks.  NDM  research  is  about  how  people  use  experience  to  make  decisions  under  time  pressure, 
shifting  conditions,  unclear  goals,  degraded  information,  and  team  interactions.  Currently,  NDM  research 
is  being  conducted  in  domains  such  as  aviation,  battle  command,  health  care,  and  process  control,  looking 
at  both  individuals  and  teams,  and  sponsored  by  all  branches  of  the  military  along  with  other  federal 
agencies  and  commercial  sources. 

The  NDM  framework  appears  to  have  great  potential  for  application  to  Army  needs.  The 
framework  addresses  the  specific  decision  requirements  (critical  and  difficult  judgments  and  decisions) 
within  a  domain  and  determines  why  these  are  difficult  and  what  cues  and  strategies  are  needed  to  handle 
the  difficulties.  This  decision-centered  approach  has  been  used  to  design  better  training  programs  and 
better  human-system  interfaces.  The  NDM  framework  also  emphasizes  training  for  team  decision 
making,  and  it  has  implications  for  better  mission  rehearsal  methods. 

During  the  10  years  in  which  it  has  been  emerging  as  a  line  of  inquiry,  NDM  research  has 
contributed  to  a  number  of  important  changes  in  decision  research.  These  include: 

•  de-emphasis  of  the  Rational  Choice  model  (because  it  cannot  be  used  in  most  field  settings), 

•  de-emphasis  of  the  heuristics  and  biases  approach  (because  the  findings  do  not  seem  to 
generalize  outside  the  laboratory), 

•  de-emphasis  of  laboratory  experiments  (because  artificial  tasks  with  naive  subjects  do  not 
result  in  useful  findings), 

•  greater  appreciation  of  expertise  in  decision  making, 

•  greater  use  of  recognitional  decision  models, 

•  growing  investigation  of  situation  awareness, 

•  modification  of  FM  101-5  (to  encourage  recognitional  strategies  along  with  analytical  ones), 

•  use  of  decision  requirements  for  training  and  design,  and 

•  increased  attention  to  team  decision  making. 

Taken  together,  these  changes  signify  a  major  shift  in  the  study  of  decision  making  and  a  strong 
potential  for  applying  NDM  findings  to  the  operational  problems  facing  today’s  Army. 
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INTRODUCTION 


The  Naturalistic  Decision  Making  (NDM)  framework  is  emerging  at  a  critical  time  for 
the  U.S.  Army.  Within  the  past  decade,  several  different  challenges  have  changed  the  importance 
of  decision  making.  First,  the  explosive  growth  of  information  technologies  created  an 
opportunity  for  the  Army  to  dramatically  improve  its  battle  command  function,  which  will 
require  changes  in  the  decision  cycle.  Second,  the  continual  downsizing  of  the  Army  will  force 
soldiers  to  broaden  their  responsibilities  to  include  greater  decision  making  at  all  levels.  Third, 
the  list  of  possible  missions  has  increased,  to  include  both  war-making  and  peacekeeping 
operations,  rather  than  the  one  overriding  mission  of  protection  from  an  assault  from  the  USSR; 
as  a  result,  commanders  are  more  likely  to  be  rushed  into  battle  with  less  knowledge  of  the 
situation. 

These  changes  add  up  to  an  Army  that  is  less  likely  to  be  following  systematic 
procedures.  The  unpredictable  implications  of  information  technologies  will  call  for  continual 
improvisation  and  adjustment.  The  downsizing  will  call  for  increased  initiative.  The  increased 
types  of  missions  will  call  for  greater  adaptability.  The  Army  may  no  longer  be  able  to  carefully 
plan  its  training  programs  and  design  its  systems  to  meet  known  requirements.  The  commanders 
and  soldiers  of  the  future  will  have  to  rely  more  heavily  on  decision  skills. 

Fortunately,  the  field  of  NDM  research  has  been  making  rapid  strides  to  meet  these 
needs.  Attempts  to  apply  traditional  decision  research,  whether  for  training  or  for  design  of 
decision  aids,  had  proven  very  disappointing.  These  efforts  continued  through  the  1970s  and 
1980s.  When  ARI  realized  that  the  traditional  approaches  were  not  going  to  meet  the  needs  of 
the  Army,  it  shifted  its  attention  to  NDM  as  a  more  promising  approach.  Eleven  years  ago,  in 
1985,  the  basic  research  program  of  the  U.S.  Army  Research  Institute  for  the  Behavioral  and 
Social  Sciences  (ARI)  began  its  investment  in  decision-making  research  that  has  emerged  as  the 
NDM  framework. 

The  support  provided  by  ARI  has  been  leveraged  into  a  major  research,  and  development 
paradigm.  During  the  period  1985-1990,  ARI  was  the  primary  source  of  funding  for  research 
projects  and  was  the  sole  sponsor  for  the  first  conference  on  NDM  (held  in  1989).  Subsequently, 
we  have  seen  a  rapid  spread  of  interest  and  application.  The  second  conference  on  NDM,  held  in 
1994,  was  sponsored  by  the  Army,  Navy,  Air  Force,  and  the  National  Aeronautics  and  Space 
Administration  (NASA)  and  featured  a  presentation  by  a  U.S.  Marine  Corps  Major  General, 
describing  the  importance  of  NDM  to  battle  command.  From  a  small  handful  of  researchers  in 
1985,  today  there  are  probably  more  than  100  active  investigators  working  in  Europe,  Asia,  and 
the  United  States. 

This  report  surveys  the  field  of  NDM  and  shows  its  potential  for  supporting  the  needs  of 
the  U.S.  Army.  The  report  is  written  from  the  perspective  of  a  researcher  who  has  been  active  in 
developing  models  and  methods  in  this  new  approach.  ^ 
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DEFINITION  OF  NATURALISTIC  DECISION  MAKING 


Naturalistic  Decision  Making  has  been  defined  as  “The  way  people  actually  make 
decisions.”  This  may  seem  simple  and  straightforward,  but  it  has  deeper  meaning  than  one  might 
expect.  The  traditional  decision  research  paradigms  that  preceded  NDM  did  not  study  what 
people  actually  did.  Instead,  these  paradigms  relied  on  artificial  laboratory  experiments  to 
investigate  the  ways  that  people  differed  from  the  optimal  solutions  arrived  at  through 
mathematical  and  statistical  analyses.  They  studied  what  people  did  not  do,  rather  than  the 
decision  strategies  they  actually  used.  This  has  limited  the  usefulness  of  traditional  studies  for 
explaining  what  happens  under  real-world  conditions. 

The  left-hand  column  of  Table  1  lists  the  central  features  of  NDM,  as  follows. 

Expertise 

In  most  settings,  novices  are  not  going  to  be  making  critical  decisions.  NDM  research 
investigates  the  way  people  use  their  experience  to  make  key  judgments  and  decisions. 
Traditional  studies  prefer  to  use  novices  because  it  is  easier  to  control  for  experience  by 
eliminating  it  and  using  novel  tasks. 

Descriptive 

Rather  than  beginning  with  an  analytical  strategy  and  testing  whether  people  use  it,  or 
trying  to  prescribe  how  people  should  be  making  decisions,  NDM  research  tries  to  understand 
and  describe  what  experienced  decision  makers  do.  The  premise  of  the  NDM  approach  is  that  we 
must  appreciate  the  strategies  people  are  using,  to  help  train  and  support  them.  Because  NDM 
research  concentrates  on  tasks  for  which  there  is  no  right  answer,  the  strategies  used  by  experts 
are  treated  as  the  standard.  In  contrast,  traditional  research  often  has  relied  on  tasks  where 
optimal  analytical  strategies  could  be  defined  using  statistical  and  logical  methods.  Because 
subjects  rarely  followed  these  strategies,  the  traditional  approach  is  prescriptive  because  it 
recommends  training  subjects  to  use  the  optimal  strategies.  If  we  insist  on  forcing  people  to  use 
unnatural  strategies,  they  either  will  become  confused  and  ineffective  or  they  will  reject  the 
support. 

Broad  Focus 

There  is  much  more  to  successful  decision  making  than  picking  the  right  option  or 
estimating  the  likelihood  of  hypotheses  based  on  the  quality  of  the  evidence.  Experienced 
decision  makers  can  size  up  situations  and  recognize  how  to  respond.  They  usually  do  not  spend 
time  comparing  options,  which  is  what  traditional  decision  research  has  focused  on.  Therefore, 
NDM  research  places  more  emphasis  on  situation  awareness  than  on  comparing  options.  It  also 
is  concerned  with  the  way  people  generate  options  and  solve  problems.  Naturalistic  settings 
usually  involve  all  these  processes.  Laboratory  researchers  prefer  to  keep  these  phenomena 
distinct  and  to  study  decision  making  in  the  absence  of  situation  awareness  and  problem  solving. 
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Table  1 

Features  of  Naturalistic  Decision  Making  Research 


Positive  Features 

Contrasts 

•  Studies  experts 

•  Studies  novices 

•  Tries  to  describe 

•  Tries  to  evaluate 

•  Takes  a  broad  focus 

•  Takes  a  narrow  focus 

Choosing  actions 

Choosing  between  alternate  actions 

Situation  awareness 

Estimating  the  probability  of  hypotheses 

Problem  solving 
•  Task  context;  field  settings 

•  Task  context:  laboratory  settings 

Time  pressure 

Ample  time 

Shifting  conditions 

Stable  conditions 

Unclear  goals 

Stated  goals 

Degraded  information 

Precise  information 

Subtle  cues  and  patterns 

Clear  inputs 

Team  interactions 

Individual  tasks 

Organizational  constraints 

Individual  tasks 

High  stakes 

Low  stakes 

•  Focuses  on  cognitive  processes 

•  Focuses  on  analytical  strategies 

•  Relies  on  Cognitive  Task  Analysis 

•  Relies  on  performance  measures 

Task  Context 

To  capture  the  task  context,  NDM  research  is  typically  performed  in  field  settings,  or  at 
the  least  in  challenging  simulations.  NDM  researchers  try  to  understand  how  people  handle  time 
pressure,  shifting  conditions,  unclear  goals,  degraded  information  (i.e.,  missing,  ambiguous,  or 
erroneous  data),  subtle  cues  and  patterns,  team  interactions,  and  organizational  constraints. 

These  are  the  defining  features  of  naturalistic  settings.  Laboratory  studies  generally  try  to 
eliminate  these  types  of  anomilies.  Traditional  decision  research  is  usually  performed  using 
artificial  tasks  under  limited-context  laboratory  conditions.  Because  traditional  researchers  want 
to  study  how  people  use  analytical  methods,  they  make  sure  subjects  have  sufficient  time. 
Shifting  conditions  make  data  hard  to  interpret,  as  do  unclear  goals,  so  traditional  decision 
researchers  tend  to  use  tasks  that  are  simple  and  stable.  Degraded  information  poses  a  problem  of 
figuring  out  how  subjects  are  interpreting  the  data,  so  it  is  easiest  to  make  sure  all  subjects  are 
clear  about  the  information  that  is  available.  For  that  reason,  subtle  cues  are  avoided  as  well. 
Teams  are  difficult  to  study  in  the  laboratory,  so  most  research  examines  individual  subjects,  free 
of  organizational  constraints. 
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Cognitive  Processes 

NDM  research  often  tries  to  understand  how  cognitive  processes  such  as  memory  and 
attention  affect  the  strategies  people  use.  In  contrast,  traditional  decision  research  often  is 
disconnected  from  cognition. 

Cognitive  Task  Analysis 

Most  NDM  studies  rely  on  methods  for  investigating  the  ways  people  are  thinking  about 
the  tasks  they  perform.  Traditional  decision  research  is  usually  confined  to  performance 
measures  of  reaction  time,  errors,  and  the  like. 

The  appeal  of  the  traditional  decision  approaches  is  that  they  are  systematic.  More 
important,  the  Rational  Choice  approach  (to  identify  alternative  options,  compare  their  strengths 
and  weaknesses,  and  choose  the  best)  offered  the  promise  of  a  set  of  general  decision  strategies 
that  could  be  trained  and  evaluated.  The  Rational  Choice  strategy  was  analytical,  so  the  basis  of 
the  decisions  could  be  articulated.  Finally,  traditional,  laboratory  research  permitted 
quantifiable  theories  and  predictions  that  could  be  tested. 

The  current  success  of  the  NDM  framework  is  largely  because  it  generated  its  own 
models  of  decision  making.  These  models  have  had  high  face  validity  and  empirical  support. 
Today,  several  models  of  NDM  exist.  One  of  the  best  known  is  the  Recognition-Primed 
Decision  (RPD)  model. 


THE  RECOGNITION-PRIMED  DECISION  MODEL 

Besides  stimulating  the  development  of  the  NDM  framework,  the  Basic  Research 
Program  of  ARI  also  sponsored  the  research  that  generated  the  RPD  model.  The  RPD  model 
described  how  people  could  use  experience  to  make  decisions  without  comparing  any  options  at 
all.  The  model  was  based  on  interviews  and  observations  of  fireground  commanders  working 
with  difficult  and  challenging  incidents.  In  the  first  such  study,  more  than  30  incidents  were 
examined,  each  containing  an  average  of  five  nontrivial  decisions.  The  commanders  were 
working  against  severe  time  pressure;  more  than  80%  of  the  decisions  were  made  in  less  than  a 
minute.  The  stakes  were  high  —  poor  decisions  could  result  in  loss  of  lives  and  property. 
Information  quality  was  uneven,  goals  shifted,  and  conditions  changed.  The  situational  dynamics 
changed  an  average  of  five  times  in  each  incident  studied.  Klein,  Calderwood,  and 
Clinton-Cirocco  (1986)  expected  that  the  commanders  would  have  to  resort  to  a  limited 
comparison  between  options  and  were  surprised  to  discover  that  the  commanders  reported  they 
were  not  making  any  comparisons  at  all. 

These  findings  raised  two  key  questions:  First,  how  could  the  commanders  be  sure  of 
carrying  out  effective  courses  of  action  without  generating  a  set  of  options  from  which  to 
choose?  Second,  how  could  the  commanders  evaluate  a  course  of  action  without  comparing  it  to 
others?  Klein  et  al.  (1986)  carefully  examined  the  interview  data  and  the  156  decision  points 
probed,  and  developed  the  RPD  model  based  on  the  fireground  commanders’  own  accounts. 
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The  answer  to  the  first  question  (how  the  commanders  did  not  have  to  generate  a  set  of 
options)  was  that  the  commanders  could  use  their  experience  to  size  up  a  situation  and  thereby 
recognize  the  typical  reaction  to  take.  They  could  generate  a  reasonable  option  as  the  first  one 
considered.  They  were  not  trying  to  find  the  optimal  solution,  but  rather  to  quickly  arrive  at  a 
workable  solution  that  could  be  enacted  in  time  to  arrest  the  spread  of  a  fire  that  might  be 
growing  exponentially. 

The  answer  to  the  second  question  (how  the  commanders  could  evaluate  an  option 
without  comparing  it  to  others)  was  that  once  the  commanders  identified  a  typical  course  of 
action,  they  would  evaluate  it  by  mentally  simulating  it  to  see  if  it  would  work  in  the  context  of 
the  situation  they  were  facing.  If  the  course  of  action  was  found  to  be  satisfactory,  it  would  be 
initiated  without  any  further  delay.  If  they  found  any  flaws,  they  would  switch  to  a 
problem-solving  mode  to  repair  the  flaws.  If  they  could  not  repair  the  flaws,  they  would  reject 
the  course  of  action  and  consider  the  next  most  typical  reaction,  repeating  the  process  until  they 
found  a  workable  option. 

The  RPD  model  is  shown  in  Figure  1 .  The  simplest  case  is  where  a  decision  maker  sizes 
up  a  situation,  forms  expectancies  about  what  is  going  to  happen  next,  determines  the  cues  that 
are  most  relevant,  recognizes  the  reasonable  goals  to  pursue  in  the  situation,  recognizes  a  typical 
reaction,  and  carries  it  out.  This  is  probably  also  the  most  common  case.  We  consider  this  a 
decision  because  reasonable  alternative  courses  of  action  could  have  been  taken.  Other  decision 
makers,  perhaps  with  less  experience,  might  have  selected  these  alternatives.  Therefore,  a 
decision  point  hypothetically  existed  even  though  the  decision  maker  did  not  treat  it  as  such. 

The  second  panel  of  Figure  1  shows  a  more  difficult  case,  where  the  decision  maker  is 
not  certain  about  the  nature  of  the  situation.  Perhaps  some  anomaly  arises  that  violates 
expectancies  and  forces  the  decision  maker  to  question  whether  the  situation  is  perhaps  different 
than  it  seems.  Another  possibility  is  that  the  uncertainty  might  be  present  from  the  beginning. 
Here,  decision  makers  must  deliberate  about  what  is  happening.  We  have  found  that  one  strategy 
they  use  is  to  build  a  story  that  explains  the  various  pieces  of  information.  If  there  are  competing 
interpretations  of  the  situation,  the  decision  maker  may  try  to  build  a  story  for  each  and  appraise 
which  story  is  the  most  consistent  and  plausible. 

The  third  panel  of  Figure  1  shows  that  once  commanders  arrive  at  an  understanding  of  a 
situation,  they  will  recognize  a  typical  course  of  action  and  then  evaluate  it  by  mentally 
simulating  what  will  happen  when  they  carry  out  the  action.  In  this  way,  if  they  spot  weaknesses 
in  their  plan,  they  can  repair  the  weaknesses  and  improve  the  plan.  This  is  probably  a  better 
strategy  than  generating  a  large  set  of  options  and  comparing  these  to  find  the  best  one.  The 
evaluation  that  uses  mental  simulation  can,  perhaps,  produce  a  better  course  of  action  instead  of 
settling  for  picking  one  from  a  set. 

The  RPD  model  hypothesizes  that  with  experienced  decision  makers: 

•  The  first  option  they  consider  is  usually  workable  so  they  do  not  have  to  generate  a 
large  set  of  courses  of  action  to  make  sure  of  getting  a  good  one. 
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Figure  1.  Recognition-Primed  Decision  Model. 


•  Comparing  options  is  not  a  goal.  They  generate  and  evaluate  options  one  at  a  time 
instead  of  comparing  their  advantages  and  disadvantages. 

•  Finding  a  workable  course  of  action  is  a  goal.  They  are  trying  to  find  the  first  option 
that  works,  not  the  best  one. 

•  Evaluating  an  option  occurs  by  imagining  how  it  will  be  carried  out,  not  through 
formal  analyses  and  comparisons. 

•  Options  can  be  strengthened  by  imagining  the  option  being  carried  out,  spotting 
weaknesses,  and  finding  ways  to  avoid  them.  (Rational  Choice  models  just  select  the 
best  without  seeing  how  it  can  be  improved.) 

•  The  focus  is  on  the  way  they  assess  the  situation  and  judge  it  as  familiar,  not  on 
choosing  between  options. 

•  The  emphasis  is  on  being  poised  to  act  quickly,  rather  than  being  paralyzed  until  all 
the  evaluations  have  been  completed. 

Since  it  was  first  proposed  in  1985,  the  RPD  model  has  received  a  great  deal  of  support. 
Klein  (1989)  has  summarized  the  data  from  studies  with  tank  platoon  leaders,  design  engineers, 
urban  and  wildland  firefighters,  and  brigade-level  military  planners,  showing  that  the  RPD 
model  accounts  for  most  of  the  decision  points  (between  50%  and  95%  of  the  decisions  made  by 
experienced  personnel),  whereas  Rational  Choice  (i.e.,  comparisons  between  options)  rarely 
occurs. 


The  initial  studies  of  the  RPD  model  have  been  replicated  several  times,  with  consistent 
support  for  the  RPD  model  and  the  general  NDM  approach.  For  example,  Kaempf,  Wolf, 
Thordsen,  and  Klein  (in  press)  have  shown  that  in  actual  incidents  of  anti-air  warfare  in  Navy 
AEGIS  cruisers,  the  Commanding  Officers  and  Tactical  Action  Officers  primarily  relied  on 
recognitional  decision  strategies;  comparison  between  options  occurred  less  than  5%  of  the  time. 
Mosier  (1990)  studied  videotapes  of  flight  crews  reacting  to  malfunctions  and  emergencies  in  a 
high-fidelity  simulator.  Pascual  and  Henderson  (in  press)  conducted  simulated  exercises  using 
experienced  officers  in  the  British  Army.  Randel,  Pugh,  Reed,  Schuler,  and  Wyman  (1994) 
conducted  simulated  exercises  with  Navy  electronic  warfare  technicians.  In  a  variety  of  domains, 
including  high  time  pressure  (urban  firefighters)  and  low  time  pressure  (design  engineers), 
individuals  (design  engineers)  and  teams  (wildland  firefighters,  commercial  aviation  crews), 
military  (Army,  Navy)  and  paramilitary  (firefighters)  and  nonmilitary  (commercial  pilots,  design 
engineers),  decision  makers  rarely  use  Rational  Choice  methods. 

Moreover,  the  RPD  process  seems  a  superior  strategy  in  naturalistic  settings.  Rational 
Choice  methods  cannot  stand  up  against  the  time  pressure,  ill-defined  goals,  and  inconsistent 
data.  The  RPD  process  can  and  does.  It  lets  the  decision  maker  short  circuit  option  comparison 
by  relying  on  experience  to  generate  a  reasonable  course  of  action  at  the  outset.  The  RPD  model 
asserts  that  people  can  use  experience  to  generate  a  reasonable  course  of  action  as  the  first  one 
considered.  Is  this  a  valid  claim?  Klein,  Wolf,  Militello,  and  Zsambok  (1995)  tested  this 
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Figure  2.  Percentage  of  first  moves  and  all  possible  moves  receiving  each  of  five  move 
quality  ratings. 


hypothesis  in  a  study  that  used  16  skilled  chess  players.  They  presented  the  chess  players  with 
difficult  positions  and  asked  them  to  think  aloud  while  trying  to  find  a  good  move.  The 
researchers  recorded  the  very  first  move  that  each  chess  player  mentioned,  despite  how  many 
they  ultimately  considered  or  which  one  they  selected.  Afterward,  the  chess  players  rated  all  of 
the  legal  moves  in  each  diagram.  Figure  2  shows  that  most  of  the  legal  moves  were  rated  as  poor 
ones.  However,  the  very  first  move  that  the  chess  players  considered  was  rated  very  high,  using 
the  players’  own  assessments.  This  means  that,  according  to  their  own  standards,  the  first  moves 
were  good  ones. 

Still,  we  might  wonder  how  good  these  first  moves  were,  by  objective  standards.  To 
investigate  this  issue,  we  used  board  positions  taken  from  games  analyzed  by  panels  of  chess 
Grand  Masters.  The  Grand  Masters  awarded  points  to  those  options  they  deemed  playable.  Table 
2  shows  that  the  Grand  Masters  awarded  points  to  only  20  out  of  124  legal  moves.  If  the 
participants  in  the  study  had  been  selecting  randomly  from  legal  moves,  they  would  have  shown 
the  same  pattern.  However,  they  showed  the  opposite  pattern.  Of  the  64  first  moves  (16 
participants  and  four  board  positions  each),  41  were  moves  that  had  received  Grand  Masters 
points,  about  two  thirds  instead  of  one  sixth.  Therefore,  using  objective  criteria,  we  find  that 
skilled  decision  makers  can  generate  good  options  as  the  first  ones  they  consider.  These  findings 
confirm  the  claim  made  by  the  RPD  model  that  people  can  use  experience  to  recognize  typical 
reactions  that  are  usually  satisfactory. 
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Klein  (1989)  also  has  speculated  about  the  boundary  conditions  for  using  recognitional 
decision  making  and  those  for  using  Rational  Choice  method.  These  are  shown  in  Table  3.  When 
time  is  short,  the  decision  makers  are  experienced,  the  conditions  keep  changing,  and  the  goals 
are  ill-defined,  a  recognitional  strategy  is  appropriate.  On  the  other  hand,  if  the  decision  makers 
are  novices,  have  ample  time,  have  to  justify  their  choice  to  others,  need  to  resolve  conflicts 
among  team  members  with  different  priorities,  have  to  find  the  best  option,  and  are  working  with 
a  task  requiring  a  great  deal  of  computational  complexity,  then  a  Rational  Choice  strategy  makes 
sense.  We  find  people  using  a  rational  choice  strategy  in  many  operational  settings,  such  as 
selecting  a  weapons  system  during  a  competitive  procurement  and  prioritizing  research  areas  to 
allocate  funds. 


Table  2 

Frequency  of  Subjects’  First  Generated  Moves  vs.  All  Legal  Moves  by 

Acceptability  Level 


First  move  generated 
by  the  subjects 

All  legal  moves 

Acceptable  moves 

41  moves 

20  moves 

(Grand  Master  points  were  awarded) 

Unacceptable  moves 

23  moves 

104  moves 

(No  Grand  Master  points  were  awarded) 

To  summarize,  the  RPD  model  was  developed  based  on  field  studies  of  the  way  that 
experienced  personnel  actually  made  decisions.  The  model  describes  how  people  can  use 
experience  to  react  rapidly  and  how  they  can  make  good  decisions  without  having  to  contrast 
options.  The  model  has  been  tested  and  supported  by  different  research  teams  working  in  a  great 
variety  of  settings.  Because  of  work  such  as  this,  we  have  a  better  idea  of  the  way  people 
actually  make  decisions. 

The  RPD  model  is  not  synonymous  with  NDM  research.  There  are  other  NDM  models. 
Moreover,  the  RPD  model  is  incomplete — it  does  not  cover  teams,  organizations,  or  issues  of 
managing  workload  and  attention,  and  so  on.  In  addition,  the  RPD  model  does  not  describe  the 
strategies  people  use  when  they  do  have  to  compare  options  in  naturalistic  settings.  The 
significance  of  the  RPD  model  is  that; 

•  it  appears  to  describe  the  most  frequently  used  decision  strategies, 

•  it  explains  how  people  can  use  experience  to  make  difficult  decisions,  and 

•  it  supports  the  observation  that  people  can  make  effective  decisions  without  going 
through  a  Rational  Choice  strategy. 
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Table  3 

Boundary  Conditions  for  Different  Strategies 


Singular 

vs. 

Comparative 

Time  Pressure 

X 

Experience  Level 

X 

Dynamic  Conditions 

X 

Ill-defined  Goals 

X 

Justification 

X 

Conflict  Resolution 

X 

Optimization 

X 

Computational  Complexity 

X 

Prior  to  the  RPD  model  and  others  like  it,  traditional  decision  researchers  speculated  that 
under  certain  task  conditions  people  would  not  use  a  Rational  Choice  strategy,  but  no  one 
presented  a  coherent  idea  of  what  the  alternative  might  be.  Most  researchers  assumed  that  it 
would  be  a  defective  version  of  the  Rational  Choice  strategy  or  some  sort  of  random  process. 
The  description  of  the  RPD  model  provided  a  firm  counter-example  to  Rational  Choice  and 
made  it  easier  for  NDM  researchers  to  take  naturalistic  decision  strategies  more  seriously. 

Cognitive  Task  Analysis  methods  have  been  essential  to  the  study  of  the  RPD  model  and 
other  NDM  processes.  Cognitive  Task  Analysis  addresses  the  cognitive  aspects  of  performing 
tasks  rather  than  the  observable  steps  and  procedures.  One  of  the  research  areas  funded  by  ARI 
in  the  mid-1980s  was  the  development  of  better  tools  for  eliciting  and  representing  knowledge. 
Cognitive  Task  Analysis  methods  had  existed  prior  to  this  time  and  were  being  developed  and 
refined  in  a  number  of  settings.  The  thrust  of  the  ARI  research  was  to  develop  Cognitive  Task 
Analysis  tools  that  could  support  research  programs  in  NDM.  Because  NDM  focuses  on  the 
perceptions  and  strategies  used  by  experienced  decision  makers,  one  of  the  keys  to  successful 
research  programs  was  to  develop  and  make  available  more  powerful  and  reliable  methods.  For 
example,  a  Critical  Decision  method  developed  and  evaluated  under  ARI  funding  concentrates 
on  difficult  and  challenging  decision  points  and  uses  these  to  probe  for  the  cues,  patterns,  and 
inferences  that  people  make.  This  method  has  been  used  in  dozens  of  different  projects  and 
domains. 

There  are  many  Cognitive  Task  Analysis  techniques,  including  interview  methods, 
observation  methods,  computer  simulation  approaches,  along  with  more  traditional  behavioral 
strategies.  The  different  Cognitive  Task  Analysis  tools  are  now  being  used  to  identify 
requirements  and  to  provide  materials  for  projects  in  training  and  in  system  design;  they  appear 
to  have  applicability  for  personnel  selection,  root-cause  analysis  and  accident  investigation, 
technology  transfer,  organizational  reengineering,  and  even  market  research. 


10 


One  of  the  drawbacks  of  most  of  the  Cognitive  Task  Analysis  methods  is  that  they  are 
labor  intensive  and  require  the  observers  and  interviewers  to  be  carefully  trained.  A  new 
generation  of  methods  is  being  designed  that  will  be  more  efficient  to  use,  although  less 
powerful  (Crandall,  Klein,  Militello,  &  Wolf,  1994).  Another  drawback  of  most  Cognitive  Task 
Analysis  methods  is  that  they  rely  on  subjective  data,  primarily  introspections  of  the  people 
being  interviewed.  People  may  distort  their  explanations  because  of  poor  memory  or  for  other 
reasons.  All  experimental  methods  have  drawbacks  that  must  be  considered.  We  have  learned  to 
use  Cognitive  Task  Analysis  data  as  a  rich  source  of  ideas,  hypotheses,  speculations,  and 
insights — an  invaluable  window  into  the  mind  of  the  decision  maker. 


THE  RANGE  OF  NATURALISTIC  DECISION  MAKING  RESEARCH 

From  1985-1990,  the  ARI  Basic  Research  Program  funded  work  by  Janet  Kolodner  from 
Georgia  Tech  on  Case-Based  Reasoning,  Raanan  Lipshitz  from  the  University  of  Haifa  on  the 
decision  strategies  used  by  Army  command  and  control  officers,  Ken  Hammond  from  the 
University  of  Colorado  on  the  decision  strategies  used  by  weather  forecasters  and  also  the  effect 
of  stress  on  decision  making,  and  Martin  Tolcott  of  Decision  Sciences  Consortium  on  ways  of 
improving  the  decision  making  of  Army  intelligence  officers,  along  with  Klein  Associates’ 
research  projects  described  previously.  It  continues  to  support  the  work  of  Len  Adelman  of 
George  Mason  University  on  the  decision  strategies  used  by  technicians  operating  Patriot  missile 
batteries.  All  these  researchers  had  been  active  prior  to  the  ARI  initiative,  and  several  of  them, 
particularly  Ken  Hammond,  had  been  identified  with  other  approaches  to  decision  research.  By 
bringing  them  together  for  annual  program  reviews,  ARI  could  gain  a  critical  mass  of 
professionals  who  had  similar  interests  and  goals,  thereby  sparking  the  NDM  movement. 

Other  researchers  had  been  working  within  the  NDM  framework  (i.e.,  the  features 
described  in  Table  1)  prior  to  1985.  Jens  Rasmussen  of  RISO  Laboratory  in  Denmark  had  been 
studying  nuclear  power  plant  operators  and  Joseph  Wohl  of  MITRE  had  been  describing  Navy 
command  and  control  officers,  to  name  just  two  prominent  examples  of  work  done  in  the  period 
1975-1985.  Jim  Shanteau  of  the  University  of  Kansas  and  his  colleagues  had  been  investigating 
proficient  decision  makers  in  a  variety  of  fields.  An  earlier  example  would  be  the  work  of 
Adriaan  de  Groot  in  1935,  who  studied  the  decision  making  of  chess  Grand  Masters.  The  NDM 
approach  did  not  come  out  of  nowhere.  Rather,  it  has  served  as  a  rallying  point  for  researchers 
who  did  not  adopt  the  traditional  approaches  to  studying  decision  making.  In  so  doing,  it  has 
focused  and  clarified  the  commonalties  among  these  efforts  and  accelerated  the  exchange  of 
ideas  and  methods. 

At  present,  NDM  research  is  being  pursued  in  a  range  of  settings  for  many  different 
sponsors.  Research  projects  can  be  classified  into  five  categories: 

•  cognitive  process  studied:  situation  awareness,  decision  making,  and/or  problem 
solving, 

•  domain  studied:  aviation,  technical  tasks,  battle  command,  health  care, 
business/industry,  process  control. 
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•  decision-making  unit  studied:  individuals  or  teams, 

•  objectives  of  the  study:  investigate,  improve  training,  improve  human-system  interface 
design,  and 

•  sponsor:  Army,  Navy,  Air  Force,  Marines,  Federal  Aviation  Administration,  NASA, 
the  National  Institutes  of  Health  (NIH),  and  others. 

The  five  categories  are  useful  for  comparing  different  projects,  and  most  projects  will  fall 
within  a  single  cell,  but  few  researchers  will  be  so  neatly  categorized.  The  Defence  Research 
Agency  of  the  United  Kingdom  (Pascual  &  Henderson,  in  press)  has  both  investigated  the 
situation  awareness  and  the  decision  making  of  British  Army  command  and  control  officers. 

ARI  has  both  investigated  and  tried  to  improve  the  performance  of  battle  command  officers,  with 
regard  to  situation  awareness,  decision  making,  and  problem  solving  (Fallesen,  1993).  The  Navy 
Personnel  Research  and  Development  Center  (Randel,  Pugh,  Reed,  Schuler,  &  Wyman,  1994) 
investigated  both  situation  awareness  and  decision  making  of  Navy  electronic  warfare 
technicians.  Stokes,  Kenper,  and  Kite  (in  press)  have  examined  the  effects  of  stress  on  both 
situation  awareness  and  decision  making  of  pilots. 

Several  researchers  are  doing  work  that  has  both  basic  and  applied  implications. 

NASA/ Ames  and  NIH  are  sponsoring  research  about  the  way  anesthesiologists  make  decisions 
in  teams  (Xiao,  Mackenzie,  and  the  LOTAS  Group,  1995;  Gaba,  1991);  the  work  has  both  basic 
and  applied  implications.  Similarly,  there  are  both  basic  and  applied  implications  to  the  work  of 
Lia  Di  Bello  (in  press),  of  the  City  University  of  New  York,  who  studied  people  working  with 
complex  software  systems.  She  has  performed  intensive  observations  of  factory  workers  using 
process  control  software  systems,  and  also  transportation  departments  relying  on  complex 
software  systems,  both  of  which  fall  within  the  domain  of  process  control.  The  Air  Force’s 
Armstrong  Laboratory  (Waag  and  Bell,  in  press)  has  established  a  research  program  on  situation 
awareness  in  F-15  pilots.  This  work  also  is  being  used  to  develop  individual  and  team  training 
for  pilots.  David  Noble  (1993;  Noble  &  Flynn,  1993),  of  Engineering  Research  Associates, 
started  with  a  series  of  investigations  into  the  decision  making  of  Navy  battle  command  officers, 
and  the  project  led  to  the  development  of  a  decision  support  system.  Mumaw,  Roth,  and 
Schoenfeld  (1993)  of  Westinghouse  have  been  modeling  the  decision  making  of  nuclear  power 
plant  operators,  to  advance  the  field  of  simulation  of  cognitive  processes,  and  also  to  provide 
better  interfaces  and  decision  support  systems. 

For  other  research  projects,  the  crossover  incorporates  work  on  individuals  and  teams. 
NASA’s  Ames  (e.g.,  Orasanu,  1990;  Mosier,  1990),  working  in  the  field  of  aviation,  is 
examining  situation  awareness,  decision  making,  and  problem  solving,  studying  both  individuals 
and  teams.  Daniel  Serfaty,  working  at  APTIMA,  studying  the  training  of  battle  command 
officers,  has  addressed  individuals  along  with  teams  (Serfaty,  Entin,  &  Volpe,  1993);  this  work 
has  been  sponsored  by  the  Army  and  Navy. 

Several  researchers  have  worked  in  more  than  one  domain.  Marvin  Cohen  of  Cognitive 
Technologies,  Inc.  has  studied  decision  making  in  commercial  flight  crews,  provided  decision 
training  to  Navy  AEGIS  officers,  and  provided  situation  awareness  training  to  Army  battle 
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command  officers  (e.g.,  Cohen,  Freeman  and  Thompson,  in  press;  Cohen  &  Freeman,  1996).  At 
Ohio  State  University,  David  Woods  and  his  colleagues’  research  spans  aviation,  health  care, 
and  process  control  (Woods,  Johannesen,  Cook,  and  Sarter,  1993).  Phil  Smith,  at  Ohio  State 
University,  examines  both  training  and  decision  support  system  for  health  care,  and  aviation. 
Eduardo  Salas  and  his  colleagues  (Jan  Cannon-Bowers,  Caroline  Prince,  Joan  Hall  Johnston)  at 
the  Naval  Air  Warfare  Center/Training  Systems  division  are  training  individuals  and  teams, 
including  AEGIS  commanders  and  aircrews  (Cannon-Bowers,  Salas,  &  Converse,  1992; 
Driskell,  Salas,  &  Hall,  1994;  Prince,  Chidester,  Cannon-Bowers,  &  Bowers,  1993;  Salas, 
Cannon-Bowers  &  Johnston,  in  press).  Researchers  are  finding  that  they  benefit  from  working 
in  several  domains  because  they  learn  from  the  similarities  and  differences  and  better  appreciate 
the  nuances  of  each  individual  domain. 

Some  NDM  researchers  and  practitioners  are  primarily  aiming  to  solve  operational 
problems.  Aer  Lingus  is  actively  presenting  individual  and  team  training  for  commercial  pilots 
(Johnston,  1992).  The  National  Fire  Academy  has  incorporated  the  RPD  model  into  training 
programs  for  fireground  commanders  (Mirabella,  Satterfield,  &  Wood,  1995),  and  was  among 
the  first  to  implement  NDM  in  training.  The  U.S.  Marine  Corps  is  using  NDM  to  improve 
training  for  battle  commanders  (Schmitt,  1996). 

Recently,  the  Navy  initiated  a  large-scale  research  program  called  Tactical  Decision 
Making  Under  Stress  (TADMUS).  This  is  an  NDM  project  studying  situation  awareness  and 
decision  making  of  individuals  and  teams  in  platforms  such  as  AEGIS  cruisers.  One  goal  of 
TADMUS  is  to  improve  the  human-system  interface;  a  second  goal  is  to  improve  individual  and 
team  training  for  decision  making  performance. 

The  interested  reader  is  directed  to  Klein,  Orasanu,  Calderwood,  and  Zsambok  (1993)  or 
Zsambok  and  Klein  (in  press)  for  additional  references.  ARI  sponsored  the  first  book  and 
cosponsored  the  second. 


FRAMEWORK  FOR  APPLYING  THE  NATURALISTIC 
DECISION  MAKING  APPROACH 

How  does  NDM  research  help  the  Army  do  its  job?  If  the  NDM  movement  does  not  offer 
direct  implications  for  improving  Army  decision  making,  then  we  must  judge  it  a  failure  no 
matter  how  reasonable  the  definition  and  features.  It  may  be  early  to  expect  an  extensive  track 
record  of  NDM  applications  but  we  should  at  least  be  able  to  predict  what  these  applications  are 
likely  to  be. 

This  section  explains  why  the  NDM  approach  is  more  likely  to  provide  useful 
applications  for  the  Army  than  traditional  decision  research.  However,  there  is  a  paradox  that 
needs  to  be  considered;  The  initial  impetus  behind  the  NDM  movement  was  to  describe  what 
people  actually  do,  whereas  the  motivation  behind  traditional  decision  research  was  to  improve 
the  way  people  make  decisions.  Therefore,  NDM  research  should  have  less  applied  value  than 
the  traditional  research,  rather  than  more. 
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The  traditional  decision  research  tried  to  identify  a  Rational  Choice  method  (generate  a 
range  of  options,  identify  evaluation  criteria,  evaluate  each  option  on  each  criterion,  calculate  the 
results,  and  select  the  option  with  the  highest  score)  that  could  help  people  make  better 
decisions.  These  steps  are  a  general  strategy  intended  to  prescribe  better  methods  than  people 
ordinarily  use.  The  classical  approaches  to  decision  making  are  centered  around  application. 
They  are  general  because  they  try  to  improve  process,  regardless  of  content  area. 

A  second  traditional  approach  to  decision  research  involves  the  demonstration  that 
subjects  in  laboratory  experiments  often  show  biases  because  of  the  way  they  use  heuristics;  e.g., 
shortcuts  in  reasoning.  This  heuristics  and  biases  approach  appears  to  have  applied  potential.  If 
we  can  describe  these  biases,  we  can  take  steps  to  train  people  to  overcome  them  or  to  build 
decision  aids  to  detect  and  alert  the  operators  to  decision  biases  or  to  take  other  steps.  These  are 
also  improvements  that  should  generalize  across  different  content  areas. 

In  contrast,  NDM  research  tries  to  describe  the  strategies  proficient  decision  makers  are 
using  and  does  not  have  any  central  claims  that  would  lead  to  implications  for  improving 
decision  quality.  It  does  not  make  sense,  for  example,  to  train  naturalistic  strategies  such  as  the 
RPD  model,  because  the  model  describes  what  people  already  do. 

There  are  several  reasons  for  expecting  NDM  research  to  result  in  applications  that  will 
improve  decision  quality  more  than  the  traditional  approaches  to  decision  making.  These  are 
shown  in  Table  4. 


Table  4 

The  Applied  Potential  of  Naturalistic  Decision  Making 


•  Classical  methods  do  not  apply  in  naturalistic  settings 

•  NDM  tries  to  support,  rather  than  replace  the  strategies  ordinarily  used 

•  Experienced  decision  makers  can  be  used  as  standards  for  performance 

•  Decision  requirements  can  be  context-specific 


Classical  methods  do  not  apply  in  many  naturalistic  settings.  Attempts  to  apply  the 
context-free  strategies  have  largely  met  with  failure.  General  strategies  must  be  weak  strategies 
because  a  one-size-fits-all  strategy  would  not  fit  any  specific  setting  very  well.  As  we  have  seen, 
the  constraints  of  naturalistic  settings  such  as  time  pressure,  inadequate  information,  shifting 
conditions,  ill-defined  goals,  and  so  forth  typically  make  it  impractical  or  impossible  to  apply 
methods  for  Rational  Choice.  Literature  reviews  have  turned  up  many  instances  of  failure  to 
train  people  to  use  Rational  Choice  strategies  (e.g.,  Howell,  1991;  Means,  Salas,  Crandall,  & 
Jacobs,  1993;  Zakay  &  Wooler,  1984).  Some  studies  have  even  shown  higher  performance  from 
subjects  using  unsystematic  strategies  than  subjects  trained  and  directed  to  use  Rational  Choice 
strategies  (e.g.,  Driskell  et  al.,  1994).  Further,  attempts  to  build  decision  aids  using  the  Rational 
Choice  model  have  been  largely  unsuccessful,  and  some  primary  sponsors  of  this  research 
during  the  past  25  years  have  admitted  their  disillusionment  (e.g.,  Tolcott,  1991).  The  decision 
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aids  were  brittle  and  became  less  helpful  as  the  problem  became  more  complex,  which  was  when 
help  was  most  needed.  When  some  of  the  conditions  listed  in  Table  1  are  relaxed,  then  we  would 
expect  that  a  Rational  Choice  method  would  be  useful.  Hammond  and  Adelman  (1976)  describe 
a  successful  use  of  a  Rational  Choice  strategy  for  selecting  a  type  of  bullet  for  use  by  the  Denver 
Police  Department,  to  satisfy  the  needs  of  police  officers  and  citizens.  Time  pressure  was  low, 
expertise  was  low,  the  problem  was  stable,  and  different  stakeholders  were  involved.  A  number 
of  Army  situations  may  be  amenable  to  Rational  Choice  methods,  even  though  most  operational 
settings  will  not  be  suited  to  Rational  Choice. 

Turning  to  the  studies  of  heuristics  and  biases,  studies  are  now  showing  that  experienced 
decision  makers  do  not  show  the  types  of  biases  found  under  the  restricted  laboratory  conditions 
(e.g.,  Christensen-Szalanski,  1986;  Fraser,  Smith,  &  Smith,  1992;  Gigerenzer,  1987;  Lopes, 

1981).  At  present,  it  is  unclear  how  much  of  an  impact  these  heuristics  have  on  less-experienced 
decision  makers.  Debiasing  methods  have  not  been  empirically  demonstrated  to  be  effective. 

When  the  decision  maker  is  experienced,  then  we  should  try  to  help  with  situation 
awareness.  When  the  decision  maker  is  inexperienced,  then  even  Rational  Choice  strategies  are 
not  likely  to  be  successful  because  naturalistic  settings  usually  do  not  allow  Rational  Choice 
strategies  to  be  used  and  because  novice  decision  makers  are  likely  to  make  errors  in  applying 
these  strategies.  Moreover,  in  many  cases,  people  wrestle  with  choices  that  have  little 
consequence.  If  decision  makers  can  easily  distinguish  which  of  two  options  is  superior,  they 
will  not  need  to  perform  the  analyses.  If  the  options  are  very  close  together  and  their  strengths 
and  weaknesses  are  well  balanced,  then  it  usually  will  not  matter  which  one  is  selected.  Decision 
makers  will  find  it  hardest  to  make  choices  as  the  differences  between  the  choices  diminishes, 
but  as  the  differences  diminish  the  implications  of  the  choice  also  diminish. 

NDM  tries  to  support,  rather  than  replace,  the  strategies  ordinarily  used.  It  is  easier  to 
help  decision  makers  better  employ  their  own  strategies  than  to  try  to  replace  these  strategies 
with  more  formal  ones.  Further,  in  most  operational  settings,  the  more  formal  and  analytical 
strategies  cannot  be  efficiently  applied,  which  would  make  decision  makers  reluctant  to  adopt  a 
new  set  of  strategies. 

Experienced  decision  makers  can  be  used  as  standards  for  performance.  NDM 
researchers  tend  to  study  the  more  experienced  decision  makers  and  try  to  identify  how  they  can 
make  effective  use  of  their  experience.  Often,  the  strategies  and  cues  used  by  experienced 
decision  makers  can  serve  as  criteria  against  which  to  measure  the  actions  of  novices.  Therefore, 
even  in  situations  where  researchers  cannot  define  the  statistical  or  logical  optimal  strategy,  they 
can  use  the  processes  found  in  experienced  decision  makers  as  performance  standards. 

Decision  requirements  can  be  context-specific.  Decision  requirements  are  the  critical 
and/or  difficult  judgments  and  decisions  within  a  task.  Investigators  can  use  Cognitive  Task 
Analysis  to  identify  these  decision  requirements.  For  each,  the  analyses  can  describe  the  reasons 
why  these  are  difficult,  along  with  the  cues,  patterns,  inferences,  and  strategies  used  by  proficient 
personnel  to  overcome  the  difficulties. 
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The  advantage  of  using  decision  requirements  is  that  they  are  context-sensitive.  They  do 
not  try  to  prescribe  a  generic  method  for  making  decisions.  Instead,  the  approach  is  to  look  at  the 
ways  that  experienced  decision  makers  make  decisions  within  their  own  domains.  Rather  than 
searching  for  general  methods,  practitioners  can  search  for  the  decision  requirements  of  the 
specific  situation.  What  matters  is  not  just  how  people  think  (their  strategies),  but  also  what  they 
think  about  (the  content). 

Decision  requirements  become  the  target  of  interventions.  If  a  battle  command  function 
is  studied  and  shows  that  inexperienced  commanders  have  difficulty  interpreting  the  intent  of 
enemy  forces.  Judging  where  to  place  reserve  units,  or  setting  priorities  for  air  assets,  these 
would  become  the  decision  requirements.  If  the  studies  identify  the  factors  that  make  these 
judgments  and  decisions  difficult,  these  would  be  highlighted  for  training  programs  (e.g.,  in 
designing  scenarios)  and  decision  support  systems  (e.g.,  for  organizing  human-system 
interfaces).  If  we  can  learn  how  proficient  commanders  work  around  the  difficulties,  we  can  use 
this  information  for  training  and  for  system  design  as  well.  Such  an  approach  would  have  higher 
face  validity  for  users,  who  would  see  the  immediate  benefits  of  the  interventions.  It  would  also 
have  greater  likelihood  for  transferring  to  the  operational  environment,  because  the  intervention 
would  be  oriented  around  the  barriers  and  difficulties  that  users  encounter  in  the  field. 

TOOLS  AND  TECHNIQUES  FOR  USING  DECISION  REQUIREMENTS 

Given  the  assumptions  described  in  the  previous  section,  the  NDM  framework  can  be 
applied  in  several  areas,  as  listed  in  Table  5.  In  covering  the  topics  presented  in  Table  5,  the 
greatest  emphasis  is  on  the  decision-centered  training  of  individuals,  because  there  are  many 
potential  interventions  that  could  quickly  be  adapted  for  use  by  Army  units.  The  other 
applications  are  also  important  though  they  are  not  covered  in  the  same  level  of  detail. 

Table  5 

Applications  of  NDM  Research 


•  Decision-centered  training  of  individuals 

•  Decision-centered  training  of  teams 

•  Decision-centered  design 

•  Mission  rehearsal 

•  Doctrine 

•  Decision-Centered  Training  of  Individuals 


Decision-Centered  Training  of  Individuals 

Decision-centered  training  of  individuals  refers  to  the  use  of  decision  requirements  in 
designing  training  programs  and  training  devices.  For  the  past  20  years,  systems  approaches  to 
training  such  as  the  Instructional  System  Design  (ISD)  procedure  have  been  dominant  in  the 
Army.  The  objective  of  ISD  is  to  take  complex  tasks  and  decompose  them  into  basic  elements, 
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specifying  the  initiating  and  terminating  conditions  for  each  element  or  subtask,  and  measure  the 
success  of  training  by  the  achievement  of  performance  criteria.  This  has  worked  fairly  well, 
despite  the  effort  needed  to  conduct  a  full-scale  ISD  study.  However,  approaches  such  as  ISD  are 
mostly  used  for  procedural  tasks  (e.g.,  setting  up  equipment)  and  not  for  judgment  and  decision 
tasks.  As  long  as  the  bulk  of  Army  training  covered  procedures,  ISD  was  satisfactory.  Now  that 
the  Army  may  need  to  train  judgment  and  decision  making,  ISD  may  need  to  be  supplemented. 
The  potential  of  NDM  work  is  to  provide  guidance  for  exactly  those  skills  where  ISD  is 
insufficient. 

Currently,  the  ARI  Research  Unit  at  Fort  Leavenworth  is  developing  a  Battle  Command 
elective  on  Practical  Thinking,  as  part  of  an  experimental  course  to  be  offered  at  the  Command 
and  General  Staff  College.  Part  of  the  impetus  for  this  work  has  been  the  recognition  of  the 
opportunities  for  focused  decision  training  generated  by  the  NDM  perspective.  The  approach 
taken  for  this  component  is  consistent  with  the  NDM  framework.  At  other  sites,  such  as  the 
National  Fire  Academy,  the  RPD  model  is  being  used  as  a  basis  for  revising  the  curriculum  to 
improve  the  decision  training.  The  U.S.  Marine  Corps  is  also  developing  strategies  for  putting 
the  RPD  model  into  action,  as  will  be  discussed. 

The  NDM  application  of  training  individuals  can  be  divided  into  decision  skills  (Table  6) 
and  training  methods  (Table  7).  Table  6  presents  the  types  of  decision  skills  that  would  be 
emphasized  within  an  NDM  framework  for  training  individuals.  None  of  these  skills  is  generic. 
Each  must  be  trained  within  the  context  of  the  operational  setting  in  which  it  is  needed. 

Table  6 

Decision  Skills  That  Can  Be  Trained 


•  Situation  awareness,  pattern  matching,  and  cue  learning 

•  Typical  cases  and  anomalies 

•  Mental  models 

•  Time  horizons 

•  PreMortem  strategy  and  Crystal  Ball  strategy 

•  Managing  uncertainty  and  time  pressure 


Situation  awareness,  pattern  matching,  and  cue  learning.  The  ARI  Research  Unit  at  Fort 
Leavenworth  is  sponsoring  applied  research  by  Marvin  Cohen  on  the  training  of  situation 
awareness.  This  training  includes  strategies  for  considering  alternative  hypotheses  and 
explanations  of  what  is  happening,  so  soldiers  become  less  likely  to  fixate  on  a  single 
explanation.  Previous  research  has  found  that  errors  may  arise  when  a  decision  maker  forms  an 
initial  hypothesis  and  then  explains  away  inconsistent  facts.  The  situation  awareness  training  is 
aimed  at  helping  officers  to  judge  when  they  are  explaining  away  too  much  so  that  they  can 
begin  to  search  for  different  explanations. 

In  using  recognitional  strategies,  decision  makers  must  be  able  to  notice  patterns,  trends, 
and  critical  cues.  These  patterns  and  cues  can  be  identified  using  Cognitive  Task  Analysis 
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methods  and  can  be  established  as  training  objectives,  using  scenarios,  simulations,  or  even 
low-level  interventions  such  as  video  clips.  Such  training  should  help  soldiers  size  up  situations 
quickly.  The  training  also  can  be  designed  to  sharpen  the  soldiers’  ability  to  detect  critical  cues 
and  to  make  fine  discriminations.  Currently,  Army  training  already  centers  around  common 
schemas  such  as  “advance  to  contact”  and  “hasty  defense.”  Each  schema  presents  its  own 
judgment  and  decision  requirements,  and  these  can  be  trained  in  conjunction  with  the  exercises. 

Typical  cases  and  anomalies.  Proficient  decision  makers  have  had  so  many  experiences 
that  they  have  learned  to  recognize  typical  cases.  The  different  interventions  listed  next  in  Table 
7  can  be  organized  to  speed  up  the  ability  to  recognize  typical  cases  and  patterns.  At  the  same 
time,  the  training  should  make  it  easier  for  soldiers  to  detect  anomalies.  These  anomalies  are 
often  the  early  warnings  to  begin  taking  precautions  or  preparing  for  contingencies. 

Mental  models.  Many  NDM  researchers  (e.g.,  Cannon-Bowers  et  al.,  1992;  Orasanu, 

1990)  are  investigating  methods  for  teaching  mental  models  to  help  trainees  see  situations  the 
way  experts  do.  These  may  be  mental  models  of  the  task,  the  equipment,  the  teamwork,  or  other 
considerations.  The  idea  of  training  mental  models  is  to  provide  a  stronger  basis  for  appraising 
situations. 

Time  horizons.  One  finding  of  NDM  research  is  that  more  proficient  soldiers  and  officers 
can  see  further  into  the  future  in  planning  their  actions.  Therefore,  we  can  establish  as  a  training 
requirement  the  design  of  scenarios,  exercises,  and  feedback  around  chains  of  events  to  enable 
trainees  to  anticipate  more  effectively.  ARI  has  taken  the  initiative  in  exploring  ways  of 
improving  the  time  horizon  of  commanders  as  a  way  of  improving  leadership  (Jacobs  &  Jaques, 

1991) . 


PreMortem  and  Crystal  Ball  strategies.  These  techniques  have  been  developed  and  used 
to  evaluate  plans  and  situation  assessments.  They  are  discussed  later  in  greater  detail,  in  the 
section  on  mission  rehearsal,  but  they  also  can  be  used  as  training  methods. 

Managing  uncertainty  and  time  pressure.  The  U.S.  Marine  Corps  has  applied  the  RPD 
model  and  the  ideas  of  NDM  to  develop  and  initiate  training  that  calls  for  rapid  tactical  situation 
assessment  and  reactions  under  uncertainty.  The  Marine  Corps  calls  this  “Quick  Decision 
Training.”  The  goal  of  this  training  is  to  develop  metacognitive  skills  so  that  trainees  can 
overcome  the  desire  for  more  information  and  gain  an  ability  to  better  manage  the  time  cycle. 

What  sorts  of  methods  would  be  promoted  by  the  NDM  framework?  Thus  far  we  have 
identified  several  interventions,  as  listed  in  Table  7.  Each  of  these  interventions  reflects  a  way  of 
using  decision  requirements  to  improve  the  way  individuals  are  trained  to  make  better  decisions. 
Each  intervention  is  a  way  of  helping  to  strengthen  the  skills  aforementioned. 
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Table  7 

Methods  for  Training  Individuals  to  Make  Better  Decisions 


•  Design  of  training  scenarios 

•  Cognitive  feedback  within  the  After-Action  Review  (AAR) 

•  Cognitive  modeling  and  expert/novice  contrasts 

•  Lists  of  common  decision  failures 

•  Test  and  Evaluation  techniques 

•  On-the-Job  Training  methods 

•  Training  device  specification 


Design  of  training  scenarios.  Decision  requirements  can  tell  us  which  judgments  and 
decisions  need  to  be  emphasized  and  which  contextual  factors  create  the  most  difficulty.  This 
information  can  be  used  to  guide  the  development  of  scenarios.  Often,  people  develop  training 
scenarios  to  present  the  right  level  of  difficulty,  without  having  clear  objectives  about  the  nature 
of  the  challenges.  By  using  decision  requirements,  scenario  developers  can  get  the  structure  they 
need.  We  must  remember  that  Practice  Training.  Simply  providing  soldiers  with  an  opportunity 
to  practice  does  not  necessarily  translate  into  better  and  more  meaningful  training.  Often, 
increased  practice  will  translate  into  skill  development.  However,  there  have  been  too  many 
instances  in  which  exercises  were  poorly  conceived  and  did  not  take  advantage  of  important 
opportunities  to  train  decision  skills.  Sometimes,  exercises  are  designed  so  carelessly  that  they 
teach  the  wrong  thing,  namely,  habits  that  will  be  dysfunctional  on  the  battlefield.  In  contrast, 
naturalistic  decision  strategies  can  be  used  to  systematically  shape  scenarios.  The  U.S.  Marine 
Corps  is  incorporating  NDM  exercises  into  the  curriculum  at  professional  schools  and  at  the 
noncommissioned  officer  level. 

Cognitive  feedback  within  the  After-Action  Review  (AAR).  After  completion  of  an 
exercise,  we  can  use  decision  requirements  to  show  soldiers  where  they  went  right  and  where 
they  went  wrong  in  making  judgments  and  decisions.  Currently,  AARs  concentrate  on  the 
specific  actions  taken.  This  is  important,  but  it  misses  the  opportunity  to  use  the  AARs  to  teach 
why  the  mistakes  were  made  so  that  the  decision  makers  can  learn  what  they  are  doing  wrong. 
We  have  observed  in  many  settings  (e.g..  Command  and  General  Staff  College,  U.S.  Army  War 
College,  armor  training  at  Fort  Knox,  rotary  wing  training  at  Fort  Campbell)  that  a  limited 
amount  of  time  is  budgeted  for  the  exercises,  leaving  little  opportunity  for  the  AAR.  Because  of 
this,  soldiers  might  learn  poor  habits  during  the  exercise,  and  get  a  chance  to  practice  these  poor 
habits,  without  ever  learning  that  they  are  doing  things  wrong.  If  the  AAR  also  covered  the 
decision  process — the  way  the  soldiers  handled  the  decision  requirements — then  the  effect  of  the 
training  exercise  could  be  greatly  leveraged  at  little  cost.  We  would  not  want  to  see  additional 
training  in  decision  making — it  is  best  to  embed  such  training  in  the  context  of  the  exercises 
already  being  run  and  to  use  the  opportunities  to  provide  feedback  about  the  decision  processes. 

Cognitive  modeling  and  expert/novice  contrasts.  Sometimes  it  is  possible  to  present 
trainees  with  information  about  how  experienced  soldiers  make  certain  types  of  decisions.  This 
can  let  the  trainees  see  what  is  possible,  what  cues  and  relationships  the  experts  notice,  and  how 
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the  experts  differ  from  the  trainees  in  the  way  they  size  up  situations.  Materials  can  include 
contrasts  between  the  way  experts  and  novices  frame  the  same  situations.  Cohen,  Freeman,  and 
Wolf  (in  preparation)  have  explored  this  strategy  in  a  project  for  the  Navy  using  Tactical  Action 
Officers,  and  we  expect  to  see  more  work  in  this  area. 

Lists  of  common  decision  failures.  We  have  found  that  some  trainees  do  not  take  training 
as  seriously  as  they  should  because  they  do  not  appreciate  the  types  of  decision  failures  that  can 
occur.  Cognitive  Task  Analysis  methods  can  document  actual  failures  in  judgments.  In  a  sense, 
the  study  of  military  history  encompasses  an  attempt  to  identify  and  learn  from  previous 
successes  and  errors.  Recent  research  has  shown  that  simply  presenting  case  histories  and 
examples  is  not  optimal  because  trainees  may  not  appreciate  what  they  are  supposed  to  conclude. 
Instead,  the  examples  can  be  placed  within  a  context  of  the  decision  processes. 

Test  and  Evaluation  techniques.  When  a  clear  set  of  decision  requirements  has  been 
developed,  it  can  be  used  to  establish  criteria  for  measuring  the  cognitive  performance  of 
soldiers.  The  decision  requirements  can  provide  a  basis  for  assessing  the  ability  of  trainees  to 
rapidly  detect  anomalies,  respond  at  the  appropriate  time  horizon,  notice  that  expectancies  have 
been  violated,  and  determine  that  events  did  not  occur. 

On-the-Job  Training  (OJT)  methods.  NDM  research  shows  that  expertise  involves  many 
subtle  aspects  of  situation  awareness.  It  is  not  always  feasible  to  incorporate  all  aspeets  into 
training  programs.  The  cost  of  schoolhouse  training  is  expensive,  and  transfer  to  the  field  is 
problematic.  Therefore,  OJT  is  likely  to  assume  greater  importance.  ARI  is  sponsoring  a  research 
program  to  identify  the  primary  aspects  of  OJT  and  to  use  these  to  increase  the  impact  of  its  own 
field  exercises.  Much  of  this  work  can  be  traced  directly  to  NDM  studies  that  showed  the 
importance  of  contextualized  practice  over  prescribed  decision  strategies.  Klein  Associates  has 
recently  studied  the  use  of  OJT  with  an  Army  National  Guard  unit. 

Training  device  specification.  The  design  of  training  devices  can  be  guided  by  decision 
requirements.  The  identification  of  decision  requirements  will  help  to  specify  the  type  of  training 
that  should  be  supported  by  a  training  device,  thereby  driving  the  device  features.  We  are 
currently  performing  a  project  for  the  Naval  Air  Warfare  Command/  Training  Systems  Division 
to  show  how  to  use  decision  requirements  to  specify  training  device  features  for  sonar  operators. 
In  addition,  decision  training  also  can  be  provided  using  extremely  low  levels  of  fidelity.  The 
Quick  Decision  exercises  used  by  the  Marine  Corps  are  paper-and-pencil  tasks  that  manage  to 
create  high  levels  of  time  pressure  and  uncertainty  and  have  received  enthusiastic  use  for 
improving  decision  skills. 

Decision-Centered  Training  of  Teams 

The  NDM  framework  has  been  useful  for  focusing  on  team  decision  making.  Traditional 
decision  research  programs  have  concentrated  on  individuals  rather  than  teams.  For  example,  the 
concept  of  shared  mental  models,  developed  and  applied  by  Orasanu  and  Salas  (1993), 
Cannon-Bowers  et  al.  (1992),  and  others,  is  being  used  in  the  Navy  to  train  teams.  Prince  et  al. 
(1993)  have  developed  several  well-accepted  team  training  programs  for  the  Navy.  Helmreich 
(1986)  has  expanded  his  initial  Crew  Resource  Management  program  from  use  in  commercial 
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aviation  to  a  wide  array  of  domains.  Zsambok,  Klein,  Kyne,  and  Klinger  (1992)  have  developed 
a  team  decision  training  module  that  has  been  institutionalized  at  the  Industrial  College  of  the 
Armed  Forces.  In  addition,  ARI  has  conducted  an  active  research  and  development  program  in 
conjunction  with  the  Command  and  General  Staff  College.  Salas  et  al.  (in  preparation)  have 
concluded  that  expert  teams  can  be  developed  by: 

•  fostering  shared  or  compatible  mental  models  of  the  task  and  of  the  roles  of  each  team 
member, 

•  training  the  team  members  on  teamwork  skills  such  as  situation  awareness,  leadership, 
and  compensatory  behavior, 

•  providing  experience  for  teams  to  function  under  the  types  of  stressful  conditions  they 
will  encounter,  by  cross-training  (letting  the  team  members  practice  on  the  roles  and 
tasks  of  others),  and 

•  showing  leaders  how  to  maintain  shared  situation  awareness. 

Decision-Centered  Design 

Decision-centered  design  refers  to  the  use  of  decision  requirements  in  designing 
information  management  systems  and  human-system  interfaces.  Currently,  the  dominant 
methods  are  a  systems-centered  design  approach  (focus  on  the  technology,  not  on  the  user’s 
needs  and  abilities)  and  a  data-centered  design  approach  (identify  all  the  relevant  information 
and  pack  it  into  the  displays). 

In  contrast,  the  NDM  approach  identifies  the  decision  requirements  in  a  given  job  or  task 
and  uses  these  in  the  conceptual  design  stage  to  help  guide  the  process.  The  impact  of 
decision-centered  design  has  been  shown  in  several  projects.  For  example,  Klinger,  Andriole, 
Militello,  Adelman,  Klein,  and  Gomes  (1993)  redesigned  the  AWACS  Weapons  Director’s 
interface.  The  new  interface  resulted  in  a  performance  improvement  of  approximately  20%  after 
only  4.5  hours  of  practice,  compared  to  the  performance  of  the  same  Weapons  Directors  using 
their  regular  interface  on  which  they  had  received  more  than  1 ,000  hours  of  practice.  Another 
successful  design  project  (Miller,  Pyle,  &  Shore,  1993)  was  for  a  decision  support  system  for 
weaponeers.  Klein  (1993)  describes  more  fully  the  use  of  decision  requirements  for  system 
design. 


Currently,  the  NDM  framework  is  being  applied  to  design  issues  within  the  context  of 
efforts  for  Digitization  of  the  Battlefield.  The  Army  Research  Laboratory  (ARL)  is  seeking  to 
use  the  NDM  approach  to  help  shape  its  participation  in  this  effort.  A  2-day  workshop  on  NDM 
was  presented  to  ARL,  and  as  a  result,  the  NDM  perspective  was  incorporated  as  part  of  a 
special  panel  reviewing  ARL  plans  for  supporting  the  Digitization  of  the  Battlefield  program. 
The  premise  is  that  the  NDM  framework  will  be  useful  in  guiding  system  design  and 
human-system  interface  efforts. 
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The  NDM  perspective  greatly  influenced  the  NATO  Research  and  Study  Group  RSG.19 
in  their  development  of  a  framework  for  Cognitive  Analysis,  Design,  and  Evaluation  (COADE) 
(Essens,  Fallesen,  McCann,  Cannon-Bowers,  &  Dorfel,  1995).  This  work,  cosponsored  by  ART, 
has  incorporated  NDM  into  concepts  for  system  design. 

The  U.S.  Marine  Corps  is  investigating  ways  to  use  NDM  and  recognitional 
decision-making  concepts  in  designing  regimental  Command  Posts.  The  objective  of  this  effort 
is  to  improve  the  battle  command  function  by  taking  advantage  of  the  findings  of  NDM  research. 

Mission  Rehearsal 

Within  the  RPD  model,  mission  rehearsal  corresponds  to  the  process  of  mental 
simulation  for  evaluating  a  course  of  action.  Research  supported  by  the  ARI  Research  Unit  at 
Fort  Rucker  showed  that  for  helicopter  crews,  mission  rehearsal  largely  consisted  of  running 
through  the  steps  and  sequence  of  the  plan,  rather  than  trying  to  anticipate  ways  that  the  plan 
might  run  into  trouble.  We  have  identified  an  active  form  of  mental  simulation  called  a 
PreMortem  exercise.  The  PreMortem  strategy  is  to  assume  that  the  plan  has  already  failed  and 
enlist  the  expertise  of  the  planners  to  explain  the  most  likely  reasons.  Instead  of  trying  to  pretend 
the  plan  is  robust,  the  planners  can  demonstrate  their  experience  and  credibility  by  finding 
weaknesses.  We  have  used  the  PreMortem  exercise  in  many  planning  settings  because  of  its 
value  in  identifying  weaknesses  at  the  outset  of  the  implementation.  Furthermore,  the  PreMortem 
strategy  has  been  expanded  as  a  mechanism  for  mission  rehearsal  by  ARI  and  was  recommended 
by  the  ARI  Research  Unit  at  Fort  Rucker,  AL,  for  use  during  Operation  Desert  Shield.  (The  lead 
time  was  not  sufficient  to  put  this  strategy  into  effect  during  the  Persian  Gulf  War.)  This  type  of 
active  mission  rehearsal  was  an  outgrowth  of  NDM  research  and  the  RPD  model  in  particular. 
Another  strategy  is  the  Crystal  Ball  method  (Cohen,  Freeman,  &  Thompson,  in  press),  which 
uses  mental  simulation  to  help  Army  commanders  consider  alternate  explanations  for  a  situation 
and  make  better  use  of  conflicting  information.  This  method  has  been  studied  at  the  Command 
and  General  Staff  College  at  Fort  Leavenworth,  KS. 

Doctrine 

One  of  the  most  important  impacts  of  NDM  research  is  on  doctrine.  Previously,  doctrine 
has  been  clear  in  promoting  the  Rational  Choice  model.  Commanders  have  long  recognized  that 
the  Rational  Choice  model  was  not  used  by  their  staff  members,  particularly  under  field 
conditions.  The  latest  version  of  Field  Manual  101-5,  Command  and  Staff  Procedures,  (U.S. 
Army,  1986)  no  longer  solely  advocates  the  Rational  Choice  model.  Instead,  several 
decision-making  approaches  and  processes  are  now  offered.  These  include  the  Rational  Choice 
model,  called  the  Deliberate  Decision  Making  process,  to  cover  conditions  in  which  analytical 
strategies  are  needed.  A  second  model  presented  in  FM  101-5  is  for  combat  conditions,  in  which 
the  mission  has  begun  and  naturalistic  conditions  preclude  the  use  of  the  Deliberate  Decision 
Making  process.  Yet  another  strategy  described  in  the  new  FM  101-5  is  for  quick  decisions,  in 
which  the  commander  and  one  or  two  staff  members  accomplish  the  planning.  A  fourth  strategy 
is  a  troop-leading  procedure,  to  help  officers  at  the  level  of  lieutenants  prepare  to  assess  the 
situation  and  respond. 
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The  use  of  these  different  strategies  is  contingent  upon  factors  such  as  experience  level 
and  the  time  available.  The  range  of  strategies  reflects  an  appreciation  for  the  need  to  enable  the 
commander  to  become  more  autonomous  during  combat  decision  making. 

The  range  of  strategies  is  consistent  with  the  need  to  establish  a  tighter  decision  cycle  and 
use  information  technologies  to  speed  the  planning  and  reaction  processes.  With  the  adoption  of 
these  new  guidelines,  the  Army  has  recognized  that  the  Rational  Choice  model  usually  slows  the 
decision  cycle  and  was  not  being  applied  effectively.  The  staff  usually  generated  only  a  single 
plausible  course  of  action  anyway  and  then  generated  a  few  “dummy”  actions  to  comply  with  the 
requirement.  This  did  not  result  in  better  plans,  but  it  did  waste  time  and  effort. 

The  U.S.  Marine  Corps  is  also  considering  the  modification  of  its  battle  command 
strategy  to  reflect  the  realities  of  recognitional  decision  making.  These  changes  are  in  reaction  to 
NDM  research.  The  Marines  are  seeking  to  adopt  a  doctrine  centered  on  decision  making  as  the 
fundamental  act  of  command,  a  doctrine  that  emphasizes  NDM  over  rational,  analytical 
strategies.  The  parallel  developments  in  the  Army  and  the  Marines  will  enable  the  two  services 
to  maintain  consistent  philosophies  and  thereby  increase  the  effectiveness  of  the  joint  operations. 

This  section  has  identified  a  range  of  methods  for  improving  Army  decision  making.  The 
basis  for  many  of  these  methods  is  the  use  of  decision  requirements.  There  are  additional  ways 
of  using  decision  requirements;  e.g.,  for  personnel  selection  (using  decision  skills  as  a  more 
naturalistic  test  of  abilities  than  other  techniques),  but  these  have  not  yet  been  closely  examined. 

Three  challenges  facing  the  Army  were  described  at  the  beginning  of  this  report.  First, 
the  growth  of  information  technologies  creates  its  own  difficulties  for  system  design  and  for 
training  soldiers  to  use  these  technologies  to  make  better  decisions.  The  NDM  framework  seems 
to  have  much  to  offer  in  helping  to  place  these  technologies  in  service  of  the  decision  makers 
rather  than  forcing  decision  makers  to  adapt  to  the  technologies.  Second,  the  downsizing  of  the 
Army  will  increase  the  responsibilities  of  soldiers  throughout  the  chain  of  command.  The  NDM 
framework  should  be  useful  in  preparing  soldiers  to  handle  these  responsibilities.  Third,  the 
unsettled  nature  of  world  events  and  the  shifting  role  of  the  military  will  increase  uncertainties 
and  make  preparedness  more  difficult.  The  NDM  framework  may  be  valuable  in  preparing  the 
Army  to  handle  these  uneertainties. 
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CONCLUSIONS 


During  the  past  10  years,  the  development  of  NDM  has  led  to: 

•  De-emphasis  of  Rational  Choice  model.  This  was  the  dominant  model  10  years  ago,  the 
source  of  guidance  in  designing  training  and  decision  aids.  Today,  military  sponsors  have 
learned  its  boundary  conditions  and  have  realized  its  limitations  for  most  field  settings. 
Although  the  Rational  Choice  model  has  not  been  rejected  for  applications  where  it  can  be 
used,  military  funding  for  research  on  this  model  has  largely  disappeared. 

•  De-emphasis  of  the  heuristics  and  biases  approach.  Although  initially  promising,  military 
sponsors  have  found  that  it  does  not  transition  out  of  the  laboratory  effectively. 

•  Confirmation  of  recognitional  decision  models.  Ten  years  ago  field  researchers  understood 
that  people  did  not  use  a  Rational  Choice  model,  but  did  not  have  alternative  accounts  of  how 
decisions  were  made.  The  RPD  and  other  naturalistic  models  provided  an  alternative  account. 
The  existence  of  these  alternative  accounts  enabled  practitioners  to  accept  that  decision 
makers  could  use  their  experience  to  recognize  courses  of  action  rather  than  analyzing  and 
comparing  them. 

•  Suspicion  of  laboratory  experiments.  Ten  years  ago  laboratory  research  was  defined  as  the 
standard  for  quality.  Today,  we  have  learned  that  many  laboratory  studies  do  not  readily 
generalize  to  operational  settings.  One  must  be  more  critical  in  accepting  and  using  their 
findings.  We  have  become  more  open  to  the  value  of  conducting  field  studies,  and  the  number 
of  researchers  working  in  naturalistic  domains  is  increasing. 

•  Appreciation  of  expert  decision  making.  Ten  years  ago,  the  traditional  research  community 
was  busy  documenting  the  failures  of  subjects  in  handling  laboratory  tasks  and  questioning  the 
capabilities  of  unschooled  decision  makers.  Today,  we  have  developed  more  respect  for 
skilled  commanders  because  of  the  models  and  studies  of  NDM.  We  have  a  better 
understanding  of  the  role  of  experience  in  decision  making.  Studies  showing  that  proficient 
decision  makers  can  quickly  generate  effective  options  have  changed  our  understanding  of 
expertise. 

•  Investigation  of  situation  awareness.  Ten  years  ago  researchers  concentrated  on  how 
subjects  compared  different  options.  Today,  situation  awareness  has  become  a  topic  of 
increasing  attention  because  we  have  realized  that  the  way  a  decision  maker  evaluates  a 
situation  is  more  important  than  the  strategies  used  to  calculate  the  best  option. 

•  Modification  of  the  doctrine  regarding  decision  making.  Ten  years  ago  FM  101-5  specified 
the  Rational  Choice  model  as  the  standard  for  military  decision  making.  Today,  FM  101-5 
encourages  the  use  of  recognitional  strategies  where  these  are  more  appropriate. 
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•  Use  of  decision  requirements.  Ten  years  ago  researchers  sought  to  use  generic  techniques 
such  as  the  Rational  Choice  model  to  improve  decision  making.  Today,  we  know  that  these 
generic  techniques  have  met  with  limited  success.  We  are  turning  to  context-driven  decision 
support  strategies,  such  as  defining  the  critical  and  difficult  judgments  and  decisions  that 
people  must  make  and  applying  these  decision  requirements  to  the  development  of  training 
programs  and  design  improvements. 

•  Emphasis  on  team  decision  making.  Ten  years  ago  most  researchers  ignored  issues  of 
teamwork  because  these  were  difficult  to  study  under  laboratory  conditions.  Today,  the  U.S. 
military  supports  an  increasing  number  of  projects  on  teamwork  and  team  decision  making, 
reflecting  the  realization  that  most  NDM  settings  involve  teams. 

•  The  decision  research  community  has  undergone  some  major  changes  during  the  past  10 
years  as  a  direct  result  of  the  NDM  framework.  Many  of  these  changes  are  dramatic  in 
their  implications.  We  are  now  equipped  with  an  array  of  tools  and  techniques  for  improving 
decision  making  in  the  types  of  field  settings  where  the  Army  and  the  other  military  services 
must  operate. 
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